M2-Project

1 Introduction and set-up

This project is made to be read in html, so open the html file in your preferred webbrowser. As standard the code is hidden in this document, but you can show all by pressing the button “Code” in the top right of the document. You can also show individual chunks of code by pressing the buttons “Code” which are placed around in the document.

Link for google colab:

Link for github: https://github.com/DataEconomistDK/M2-Group-Assignment

R-rules: (delete before handin) Data may not be overwritten, but may always be given new meaningfull names. data_raw, data_tidy ect. Always lower case letters. Inline code comments are only for technical use ect. All other explanation should be text. We load all packages in the top. We write in document without any branches.

In this project we will work with a dataset of 5.000 consumer reviews for a few Amazon electronic products like f. ex. Kindle. Data is collected between September 2017 and October 2018. This is a sample taken from Kaggle which is a part of a much bigger dataset available trough Datafiniti. The data can be collected from this link: https://www.kaggle.com/datafiniti/consumer-reviews-of-amazon-products?fbclid=IwAR1o_blPfHeBPmnUzAOW7Ct24L7fhbI3OGcbfaVgaDZENhVXwaCP4godKvQ#Datafiniti_Amazon_Consumer_Reviews_of_Amazon_Products.csv

Note there is 3 available dataset on kaggle, but the file used here is called “Datafiniti_Amazon_Consumer_Reviews_of_Amazon_Products”. The file is downloaded as is, and imported further below.

1.1 Loading packages

First i have some personal setup in my local R-Markdown on how i want to display warnings ect. And then i load my packages.

### Knitr options
knitr::opts_chunk$set(warning=FALSE,
                     message=FALSE,
                     fig.align="center"
                     )

options(warn=-1) # Hides all warnings, as the knitr options only work on local R-Markdown mode. 

Sys.setenv(LANG = "en")

# Packages

if (!require("pacman")) install.packages("pacman") # package for loading and checking packages :)
pacman::p_load(knitr, # For knitr to html
               rmarkdown, # For formatting the document
               tidyverse, # Standard datasciewnce toolkid (dplyr, ggplot2 et al.)
               data.table, # for reading in data ect. 
               magrittr,# For advanced piping (%>% et al.)
               igraph, # For network analysis
               tidygraph, # For tidy-style graph manipulation
               ggraph, # For ggplot2 style graph plotting
               Matrix, # For some matrix functionality
               ggforce, # Awesome plotting
               kableExtra, # Formatting for tables
               car, # recode functions 
               tidytext, # Structure text within tidyverse
               topicmodels, # For topic modelling
               tm, # text mining library
               quanteda, # for LSA (latent semantic analysis)
               uwot, # for UMAP
               dbscan, # for density based clustering
               SnowballC,
               wordcloud,
               textdata,
               wordcloud, 
               textstem, # for textstemming 
               tidyr,
               widyr
               )

# I set a seed for reproduciability
set.seed(123) # Have to be set every time a rng proces is being made. 

1.2 Loading and filtering data

Now we load the data we downloaded from kaggle. From this file we select the following variables:

• id: An id number given to each review created by us corrensponding to the row number of the raw data.

• name: The full name of the product

• reviews.rating: The rating of the product on a scale from 1-5.

• reviews.title: The title of the review, given by the customer.

• reviews.text: The review text written by the customer.

The data is cleaned by removing punctuations, numbers and changing all strings to lower case in the review text.

data_raw <- read_csv("Datafiniti_Amazon_Consumer_Reviews_of_Amazon_Products.csv") %>% 
  select(name, reviews.rating, reviews.text, reviews.title) %>% 
  mutate(id = row_number())

data_clean <- data_raw %>% 
  mutate(reviews.text = reviews.text %>% str_remove_all("[[:punct:]]")) %>% 
  mutate(reviews.text = reviews.text %>% str_remove_all("[[:digit:]]")) %>% 
  mutate(reviews.text = reviews.text %>% str_to_lower())

We here want to primarily work with tidy text, where there is one token per row. This is now created below.

tokens_raw <- unnest_tokens(data_clean, word, reviews.text)
tokens_clean <- tokens_raw # extra filtering at some point? 

We now have 153.085 tokens, in their each seperate rows.

2 Network analysis

In this assignment we want to use network analysis to gain new insights into how the reviews are structured. Here we extract bigrams from each review text, clean and prepare them to then create networks. Where we before considered tokens as individual words, we can create them as n-grams that are a consecutive sequence of words. Bigrams are n-grams with a length of 2 consecutive words. This can be used to gain context and connection between words. These are now created. Then the most common bigrams are displayed.

bigrams <- data_clean %>%
  unnest_tokens(bigram, reviews.text, token = "ngrams", n = 2) # n is the number of words to consider in each n-gram. 

bigrams$bigram[1:2]

## [1] "the display" "display is"

Remember that each bigram overlap, as can be seen from above, so that the first token is “the display” and the second is “display is”.

#Counting common bigrams
bigrams %>% 
  count(bigram, sort = TRUE)

Notice the most common bigrams are: “for my”, “easy to”, “to use”, “it is”. These are mostly stopwords, which is not very usefull for the analysis. To remove these from the bigrams, we now split the bigram into 2 columns word1 and word2, and then filter them away if either of them is a stopword. The stopwords are taken from a dictionary called stop_words. Now we make a new count to see the most bigrams.

bigrams_separated <- bigrams %>% 
  separate(bigram,c("word1","word2"),sep = " ")

bigrams_filtered <- bigrams_separated %>% 
  filter(!word1 %in% stop_words$word) %>% 
  filter(!word2 %in% stop_words$word)

#New bigram counts
bigram_counts <- bigrams_filtered %>% 
  count(word1, word2, sort = TRUE)

bigram_counts

Above we can see that the most common bigrams are now mostly product names such as “kindle fire”, “battery life”, “amazon fire”, “amazon echo” ect. We now combine the 2 columns again into a single column with the bigram, to do further analysis.

bigrams_united <- bigrams_filtered %>% 
  unite(bigrams, word1, word2,sep = " ")

bigram_tf_idf <- bigrams_united %>%
  count(reviews.title, bigrams) %>%
  bind_tf_idf(bigrams, reviews.title, n) %>%
  arrange(desc(tf_idf))

bigram_tf_idf

bigram_graph <- bigram_counts %>% 
  filter(n > 15) %>% 
  graph_from_data_frame()



a<- grid::arrow(type = "closed",length = unit(.15,"inches"))

ggraph(bigram_graph, layout = "fr") + 
  geom_edge_link(aes(edge_alpha = n), show.legend = FALSE,
                 arrow = a, end_cap = circle(.05,'inches'))+
  geom_node_point(color = "pink", size = 3) + 
  geom_node_text(aes(label = name),vjust=1,hjust=1) + 
  theme_void()

#Correlation bigrams
bigram_section <- data_clean %>% 
  unnest_tokens(word,reviews.text) %>% 
  filter(!word %in% stop_words$word)
bigram_section

word_pairs <- bigram_section %>% 
  pairwise_count(word, id, sort = TRUE)

word_pairs

word_cors <- bigram_section %>% 
  group_by(word) %>% 
  filter(n()>= 20) %>% 
  pairwise_cor(word,id,sort=TRUE)

word_cors

word_cors %>% 
  filter(item1 %in% c("kindle","fire")) %>% 
  group_by(item1) %>% 
  top_n(6) %>% 
  ungroup() %>% 
  mutate(item2 = reorder(item2, correlation)) %>% 
  ggplot(aes(item2, correlation)) +
  geom_bar(stat = "identity") +
  facet_wrap(~ item1, scales = "free") +
  coord_flip()

word_cors %>% 
  filter(correlation > .275) %>% 
  graph_from_data_frame() %>% 
  ggraph(layout="fr") +
  geom_edge_link(aes(edge_alpha = correlation), show.legend = FALSE) +
  geom_node_point(color = "pink",size=3) +
  geom_node_text(aes(label = name), repel = TRUE) +
  theme_void()

tokens_clean %>% 
  count(word,sort =TRUE)

options(max.print = 10)
unique(tokens_clean$word)

##  [1] "i"       "thought" "it"      "would"   "be"      "as"      "big"    
##  [8] "small"   "paper"   "but"    
##  [ reached getOption("max.print") -- omitted 5935 entries ]

unique(tokens_clean$word)

##  [1] "i"       "thought" "it"      "would"   "be"      "as"      "big"    
##  [8] "small"   "paper"   "but"    
##  [ reached getOption("max.print") -- omitted 5935 entries ]

#Lemmatazion
tokens_lemma <- tokens_clean %>% 
  mutate(word = lemmatize_words(word))

unique(tokens_lemma$word)

##  [1] "i"     "think" "it"    "would" "be"    "as"    "big"   "small"
##  [9] "paper" "but"  
##  [ reached getOption("max.print") -- omitted 4539 entries ]

#Stemmed
tokens_stemmed <- tokens_clean %>% 
  mutate(word = wordStem(word))

unique(tokens_stemmed$word)

##  [1] "i"       "thought" "it"      "would"   "be"      "a"       "big"    
##  [8] "small"   "paper"   "but"    
##  [ reached getOption("max.print") -- omitted 4329 entries ]

3 NLP

tokens_clean %>% count(word, sort=TRUE) %>% head(100)

tokens_clean %<>%
  anti_join(stop_words)

own_stopwords <- tibble(word= c("im", "ive", "dont", "doesnt", "didnt"), 
                        lexicon = "OWN")

tokens_clean %<>% 
  anti_join(stop_words %>% 
              bind_rows(own_stopwords), by = "word") 

tokens_stemmed <- tokens_clean %<>%
  add_count(id, word, name = "nword") %>%
  add_count(id, name = "ntweet") %>%
  filter(nword > 1 & ntweet > 1) %>%
    select(-nword, -ntweet)

tokens_stemmed <- tokens_stemmed %>% 
  mutate(word = wordStem(word))

topwords <- tokens_stemmed %>% count(word, sort=TRUE)

topwords %>%
  top_n(20, n) %>%
  ggplot(aes(x = word %>% fct_reorder(n), y = n)) +
  geom_col() +
  coord_flip() +
  labs(title = "Word Counts", 
       x = "Frequency", 
       y = "Top Words")

topwords %>%
  ggplot(aes(x = n)) + 
  geom_histogram()

wordcloud(topwords$word, topwords$n, random.order = FALSE, 
          max.words = 50, colors = brewer.pal(8,"Dark2"))

3.1 TF-IDF

• what is tf-idf
• comment on not running a tf-idf

##Sentiment analysis

###Bing

sentiment_bing <- tokens_stemmed %>% inner_join(get_sentiments("bing"))
sentiment_bing %>% count(sentiment)

sentiment_analysis <- sentiment_bing %>% filter(sentiment %in% c("positive"
, "negative"))

word_counts <- sentiment_analysis %>%
count(word, sentiment) %>%
group_by(sentiment) %>%
top_n(10, n) %>%
ungroup() %>%
mutate(
word2 = fct_reorder(word, n))

ggplot(word_counts, aes(x = word2, y = n, fill = sentiment)) +
geom_col(show.legend = FALSE) +
facet_wrap(~ sentiment, scales ="free") +
coord_flip() +
labs(title ="Sentiment Word Counts",x ="Words")

tokens_stemmed %>% inner_join(get_sentiments("bing")) %>% count(reviews.rating, sentiment)

tokens_stemmed %>%
inner_join(get_sentiments("bing")) %>%
count(reviews.rating, sentiment) %>%
spread(sentiment, n) %>%
mutate(overall_sentiment = positive - negative)

###Afinn

sentiment_afinn

ggplot(sentiment_afinn,aes(x = reviews.rating, y = sentiment, 
                           fill = as.factor(reviews.rating))) +
geom_col(show.legend = FALSE) +
coord_flip() +
labs(title ="Overall Sentiment by Stars",subtitle ="Reviews for Robotic Vacuums",
     x ="Stars",y ="Overall Sentiment")

labs(title ="Sentiment Word Counts",x ="Words")

## $x
## [1] "Words"
## 
## $title
## [1] "Sentiment Word Counts"
## 
## attr(,"class")
## [1] "labels"

##LSA

##LDA

4 Machine learning